Generating Super Resolution Digital Images 



Field of the invention: 

The present invention relates to digital images and more particularly to the 
acquisition and processing of digital images. 

Background of the Invention: 

The technology to detect and read digital watermarks that are embedded in images 
is well developed. For example see, U.S. patent 5,721 ,788, U.S. patent 5,745,604, 
U.S. patent 5,768,426 and U.S. patent 5,748,783. Programs for detecting and 
reading digital watermarks are included in various commercially available image 
editing programs such as Adobe Photoshop that is marketed by Adobe Corporation. 

A digital watermark can be more easily detected and read from a high quality, high 
resolution image, than from a low quality or low resolution image. In some 
situations multiple images having similar picture content are available. There are 
known techniques for combining multiple low resolution images which have similar 
content in order to make one relatively high resolution image. Such a technique is, 
for example, described in US Patent 6,208,765. The system shown in patent 
6,208,765 aligns images using a reference coordinate system. An enhanced 
images is then synthesizes and regions of image overlap (i.e. regions of similar 
image content in multiple images) have improved quality. The synthesis process 
combines information in overlapping regions to form an enhanced image that 
corrects many of the image impairments. 

Inexpensive low resolution cameras designed for connection to personal computers 
are in widespread use. Such camera are herein referred to as PC cameras. PC 
cameras generally capture pixels in what is often termed a "Bayer pattern". A Bayer 
pattern is a four pixel square where only one color is captured for each pixel. The 
colors captured for the two pixels on the first line are red and green. The colors 
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captured for the two pixels on the second line are green and blue. Interpolation is 
used to calculate three colors for each pixel position. The positions in the Bayer 
pattern where the value of a color is calculated rather than actually measured are 
herein termed "holes". 

If a camera which uses pixel interpolation is used to acquire a digital image of a 
watermarked physical image, the pixel interpolation may make it more difficult to 
accurately read the watermark from the acquired digital image. However, with 
camera such as PC cameras, it is easy to obtain multiple images which have almost 
identical content. The present invention is directed to using such multiple images to 
minimize or eliminate the need to interpolate to obtain a high resolution image. 

Summary of the Present Invention: 

The present invention is directed to producing a high resolution image from multiple 
images which have similar content. Where a camera such as a PC camera is used 
to acquire a digital image, in general, the camera will have slightly moved between 
when successive images are captured. With the present invention, such slight 
camera movement between when successive images are captured is 
advantageously utilized to minimize or eliminate the need to interpolate in order to 
fill in the "holes" in a Bayer pattern. 

With the present invention, the captured color values from multiple appropriately 
positioned images are used to fill in the "holes" in a Bayer pattern. For example, 
instead of interpolating the value of red for the second pixel position on the first row 
of a Bayer pattern, an image is selected which is positioned one pixel to the right of 
the first image, and the red values from this image are used for the red values of the 
second pixel on the first line. Furthermore, the value of the pixels in multiple images 
which are appropriately aligned to each pixel position can be averaged to generate 
a better value for each pixel position. 
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With the present invention, information carried by a digital watermark (either alone 
or together with other techniques) is used to determine the alignment of the images. 
Images are selected which are positioned so that corresponding pixels fall within a 
specified tolerance from a location in a Bayer pattern. That is, images are selected 
that are within a specified tolerance of one pixel to the right or one pixel down from 
a reference frame. The pixel values of the images which fall within the specified 
tolerance of each pixel position in a Bayer pattern are selected and combined to 
form a high resolution image. 

Brief Description of the Drawings; 

Figure 1 illustrates a system for capturing multiple images which have similar 
content. 

Figure 2 illustrates the Bayer patterns in an image. 

Figure 3 illustrates how four low images can be combined to fill in the holes in a 
Bayer pattern without using interpolation. 

Figure 4 is a flow diagram illustrating the operation of the invention. 
Detailed Description: 

The first preferred embodiment of the invention utilizes the invention to facilitate 
reading digital watermarks from images captured by an inexpensive camera that is 
connected to a personal computer. Figure 1 is an overall diagram of the system 
used to practice the first embodiment of invention. 

The system shown in Figure 1 includes a camera 101 connected to a personal 
computer 102. The computer 102 has a storage system 102A that stores programs 
and images. The camera 101 is directed at a physical image 105. The physical 
image 105 includes a digital watermark. The watermark could for example have 
been embedded in image 105 using the commercially available image editing 
program Adobe Photoshop. As is conventional with watermarks embedded with the 
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Adobe Photoshop program, the digital watermark embedded in image 105 includes 
a "grid signal" and a "payload" signal that carries digital data. 

Watermark reading programs, such as that included in the Adobe Photoshop 
program, use the grid signal tp align and scale a captured image prior to reading the 
payload data from the watermark. In the frequency plane, (i.e. when the frequency 
of the grid signal is examined) the grid signal forms a recognizable pattern. The 
location and shape of this pattern indicates the rotation and scale of the image. 
When the image is adjusted to the correct rotation and scale, the size and location 
of the "watermark tile" (i.e. the redundant pattern in the image that carries the 
watermark) is such that watermark payload signal can be easily read. 

The camera 101 can for example be the camera marketed by the Intel Corporation 
under the trademark "Intel PC Camera Pro Pack" Such a camera is relatively 
inexpensive and it produces an image with a 640 by 480 resolution. The camera 
has detectors positioned in a 640 by 480 configuration; however, each detector only 
captures one color. The color captured by each detectors is that specified by a 
Bayer pattern. Figure 2 illustrates how colors are captured in a Bayer pattern. 
There is a "hole" for each color not captured at a particular location. In the prior art, 
interpolation is used to determine the values of the colors for the "holes" in the 
Bayer pattern. With the present invention interpolation is not used to fill in the holes 
in the Bayer pattern. 

It is possible to read a watermark from an image captured by a camera when 
interpolation is used to fill in the holes in a Bayer pattern. However, when 
interpolation is used to fill in the holes in a Bayer pattern, the camera must be 
correctly positioned (i.e. within a relatively small tolerance) with respect to the image 
and in some situations, several attempts to read an image may be required. The 
present invention is directed to making it easier to read digital watermarks from 
images captured by a relatively low resolution camera. 
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The conventional PC camera 101 can capture individual images or it can capture 
multiple images at a rate of up to 30 frames per second. The camera 101 is 
controlled by a computer program. With the present invention, values from multiple 
images are used to fill in the holes in a Bayer pattern to create a relatively high 
resolution image. 

Figure 3 illustrates (in a greatly exaggerated form) how the red color from four 
relatively low resolution images 301 to 304 can be combined into the red color for 
one relatively high resolution image. The red pixels in image 301 are represented by 
outline circles, the red pixels in image 302 are represented by outline squares, the 
red pixels in image 30 are represented by solid circles and, the red pixels in image 
304 are represented by solid squares. Only the red pixels (i.e. the pixels in the 
upper left hand corner of a Bayer square are shown in Figure 3. It is should be 
understood that the other pixels are handled in a similar manner. Furthermore, 
Figure 3 only shows a small number of pixels, naturally in an actual image there 
would be many such pixels. 

The four images 301 to 304 are combined as indicated by the alternating squares 
and circles in image 305, In order for the process to produce a useful result, the 
images must be aligned, so that corresponding pixels from the various images are 
next to each other, one pixel to the right and/or one pixel down as shown in Figure 
3. The alignment must be within a certain tolerance which in this embodiment is 
one tenth of a pixel width. If the initial images have a resolution of 640 by 480 as 
produced by the Intel PC camera, and if the image is ten inches square, the pixels 
must be aligned to the locations in a Bayer pattern to within 0.012 inches. A very 
slight movement of the camera which captured the images could produce images 
so positioned. 
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With the present invention, the camera 101 is used to capture multiple images. For 
example in one second it can capture 30 images. The images are captured at a 
high frame rate so that the relative location of the physical image 105 and the 
camera are substantially (but not exactly) the same for all images. 

As an example, consider the red pixel in a Bayer square and consider a 
corresponding pixel (herein called the reference pixel) in each of the 30 images 
captured during a one second interval. With the present invention the 30 images 
are divided into five categories, (for reference the four positions in a Bayer Square 
are herein referred to as positions 1 to 4). 

1) Those images within 0.1 pixel of position 1 of the Bayer square. 

2) Those images within 0.1 pixel of position 2 of the Bayer square. 

3) Those images within 0.1 pixel of position 3 of the Bayer square. 

4) Those images within 0.1 pixel of position 4 of the Bayer square. 

5) The remaining images. 

The pixel values in the sets of images designated 1, 2, 3, and 4 above are averaged 
generating four images that will be termed the four "averaged" images. 
The four averaged images are combined into one image as indicated in Figure 3. 
That is, images 301 to 304 represent four averaged images. 

In some situations, there may not be images found which are located in each of the 
desired positions. If there are no images in one of the categories, the other 
averaged images are combined and the fourth pixel position is determined by 
interpolated in accordance with the prior art. 

Figure 4 is a block diagram of a computer program which performs the operations of 
the present invention. As indicated by block 401, a series of images are captured 
with a PC camera. For example thirty images could be captured over a one second 
period. The operator will try to hold the camera such that the relative position of the 
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camera and the printed image remain constant; however, there will almost always 
be some movement. Note, that the amount of movement that is relevant to this 
invention is the size of a pixel. 

Next the watermark grid signal is read from each image and the relative position of 
each image is determined. As indicated by block 403, the images are divided into 
five categories as follows: 

1) Those images within 0.1 pixel of position 1 of the Bayer square. 

2) Those images within 0.1 pixel of position 2 of the Bayer square. 

3) Those images within 0.1 pixel of position 3 of the Bayer square. 

4) Those images within 0.1 pixel of position 4 of the Bayer square. 

5) The remaining images. 

Next as indicated by block 404, the pixel values from the images in each of the first 
categories are averaged to generate four images with average pixel values. 
The four images with average pixel value are next combined into one image as 
indicated by block 405. The combination is as shown in Figure 3. 

If any holes remain in the Bayer blocks, these holes are filled in by interpolation in 
accordance with the prior art as indicated by block 406. The above described how 
the "red" color for each pixel in the high resolution image is determined. The blue 
color for each pixel is determined in a similar manner. The green pixels are also 
handled similarly; however, it is noted that for the green color there are two acquired 
pixels in each Bayer square, thus, there are less "holes" in the green color. 

Finally, as indicated by block 407, the watermark payload data is read from the 
combined image in a conventional manner. 

It is noted that in the first embodiment of the invention, a conventional watermark 
grid signal is used to align the images. In alternate embodiments, any reference 
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signal which is inserted into the image can be used for alignment. For example a 
pseudo random noise pattern with good correlation properties or fiducial marks of 
some kind can be used. Preferably, the reference signal added to an image should 
not be visible to the human eye. 

It is also noted that in the first embodiment described above only a watermark grid 
signal is used to align the images. In alternate embodiments, the alignment 
technique described herein can be used together with other known image alignment 
techniques, such as correlating image content, to align the images. Thus both a 
hidden reference signal as described with reference to the first embodiment of the 
invention and image content can be used to align images. The image content 
would be used to align the images as described in the prior art. The use of a 
combination of techniques in some situations will produce better alignment than the 
use of a single alignment technique. 

In the embodiment shown, the images are combined in accordance with the 
positions of a Bayer square. It should be understood that other color patterns and 
other patterns of positions could be used in alternate embodiments. 

While the invention has been shown and described with respect to preferred 
embodiments thereof, it should be understood that wide variety of changes in form 
and design can be made without departing from the spirit and scope of the 
invention. The scope of the invention is limited only by the appended claims. 
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